[MySQL] Fix replication stalls caused by silently dropped idle connections - #760
Conversation
🦋 Changeset detectedLatest commit: a6370df The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
rkistner
left a comment
rkistner
left a comment
There was a problem hiding this comment.
To clarify, this specifically enables TCP keepalive, not a mysql-specific keepalive?
Is there an option on MySQL / Zongji to additionally do a mysql-level keepalive? If there is no built-in functionality for that, maybe something along the lines of a SELECT 1 query every minute?
Your work here is definitely an improvement, but the firewall silently dropping the connection due to being idle is not the only case this can happen, and TCP keepalives typically take very long to detect the issue on their own. A protocol-level keepalive with a socketTimeout or equivalent can be a much stronger check.
Yes the change was TCP The service now creates the control connection itself and passes it into I'll also open a small types PR to |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3edb4060bb
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
rkistner
left a comment
rkistner
left a comment
There was a problem hiding this comment.
Looks good to me; just one comment on connection error handling.
It would be nice to get the relevant types upstream in our @powersync/mysql-zongji package, but it's not a blocker here.
Co-authored-by: Ralf Kistner <ralf@journeyapps.com>
Rentacookie
left a comment
Rentacookie
left a comment
There was a problem hiding this comment.
LGTM - thank you for handling the cleanups! 🙏
3 participants
Summary
The PowerSync Service keeps two connections open to MySQL: one that streams changes, and a helper connection used for occasional lookups. The helper can sit idle for hours. Some firewalls quietly close connections that have been idle for an hour. The next time the PowerSync Service tries to use that closed connection, it waits ~15 minutes for the operating system to give up. During that wait replication is completely frozen, nothing is logged, and the PowerSync Service still reports itself as healthy. Shutdown has the same problem, because it sends one last query over that same connection and waits for the answer with no time limit. A self-hosted customer hit this issue five times in two days and traced it to their firewall's one hour idle timeout.
Fix
The MySQL connections now send a
keepalivesignal every 40 seconds. That is enough for a firewall to see the connection as active, so it never gets closed in the first place. Shutdown also gets a 5 second limit: if the final query gets no answer in that time, the service closes the connection itself instead of waiting.AI disclaimer
I developed this change using Claude Opus 4.8 for the investigation and Claude Fable 5 for the implementation. I reviewed and tested the changes myself.